Ergodic Properties of weak Asymptotic 
Pseudotrajectories for Set-valued Dynamical 

Systems 

Mathieu Faure* and Gregory Roth^ 

Abstract: A successful method to describe the asymptotic behavior of vari- 
ous deterministic and stochastic processes such as asymptotically autonomous 
differential equations or stochastic approximation processes is to relate them to 
an appropriately chosen limit semiflow. Benai'm and Schreiber (2000) define a 
general class of such stochastic processes, which they call weak asymptotic pseu- 
dotrajectories and study their ergodic behavior. In particular, they prove that 
the weak* limit points of the empirical measures associated to such processes 
are almost surely invariant for the associated deterministic semiflow. Continu- 
ing a program started by Benai'm, Hofbauer and Sorin (2005), we generalize the 
ergodic results mentioned above to weak asymptotic pseudotrajectories relative 
to set-valued dynamical systems. 
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1. Introduction 

Let {E, d) be a complete metric space and consider a flow (j) on namely (j) '■ -ExM — > 
E is continuous and satisfies the following properties: 

(i) for any x G E, (j){x, 0) = x, 

(a) for any t, s G M and x & E, (j){(j){x, t),s) = (/>(x, t + s). 

In the sequel, we will prefer the notation (ptix) instead of <f>{x,t). A continuous 
function X : M+ — > E is an asymptotic pseudotrajectory (APT) for the flow (j) if 

lim sup d{X{t + s),<l)siX{t))) =0, (1) 

*^°°sG[0,T] 

for any T > 0. Heuristically this means that, for any T > 0, the curve joining X{t) 
to X{t + T) shadows the trajectory of the solution starting from X{t) with arbitrary 
accuracy, provided t is large enough. This concept has initially been introduced in 
Benai'm (1996) and Benai'm and Hirsch (1996), where the authors proved that the 
asymptotic behaviors of an APT can be described with a great deal of generality 
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through the study of the asymptotics of the flow (p. One of their main results is 
the characterization of the limit set of an APT, in the sense that it is internally 
chain transitive, i.e. compact, invariant and contains no proper attractor for the 
restricted flow (the terminology comes from the notion of chain recurrence introduced 
by Conley (1978), see also Bowen (1975)). 

Consequently, this result turns out to be a particularly useful tool for analyzing 
the long term behavior of a large class of "perturbed" systems, whose solutions are 
APTs relative to some "unperturbed" flow. For instance, given an asymptotically 
autonomous differential equation, its solution trajectories are APTs relative to the 
flow induced by its limit autonomous differential equation (see Benai'm and Hirsch 
(1996)). Also, under the right assumptions, the paths of a stochastic approximation 
process with decreasing step size are almost surely APTs for the flow induced by the 
mean ODE (see e.g. Benaim (1999) or Pemantle (2007) for comprehensive overviews 
on the topic). 

In Benaim and Schreiber (2000), the authors investigate the ergodic or statisti- 
cal behavior of APTs for a flow. In fact, they prove their main result for a more 
general class of stochastic processes that they call weak asymptotic pseudotrajecto- 
ries (WAPT). Given a probability space (f2,3", P) and a nondecreasing family of 
sub-(T-algebras (3'i)t>o, a process 

X : M+ X 17 — > E, 

is called a weak asymptotic pseudotrajectory (WAPT) for (j) provided that 

(i) It is progressively measurable : X|[o,T]xn is 'B[o,t] x9"t measurable for all T > 0, 

(ii) for each q > and T > 0, 

lim P( sup d(X{t + s), 0,(X(t))) > a| Jt) = 
se[o,r] 

almost surely. 

Notice that, if a random process X is almost surely an APT, namely for almost 
every ui G fl, (1) holds, then X is a weak asymptotic pseudotrajectory. One should 
be aware that the characterization of limit sets no longer holds for a WAPT. By this 
we mean that they are not almost surely internally chain transitive in general. One 
of the main result of the paper quoted above is the following 

Theorem A (Benaim and Schreiber (2000), Theorem 1) Given a WAPT X for a 
flow (j), the weak* limit points of the empirical measures 

1 /•* 

l^t{^) ■■= - / Sx{s,u)ds 

<^ Jo 

(where 6p is a Dirac measure at the point p) are almost surely invariant measures 
for (j). In particular, if the family of measures { j /q 5x(s)<^s}t>0 is tight, this implies 
that, with probability one, the process X(t) spends most of its time in any arbitrarily 
small neighborhood of the Birkhoff center of (p. 
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In Benai'm, Hofbauer and Sorin (2005), the authors generahzed the notion of asymp- 
totic pseudotrajectory to set-valued dynamical systems $ (induced for instance by 
a differential inclusion, see Section 2 for a general definition of set- valued dynamical 
systems and section 4 for the particular case of differential inclusions) and extended 
the characterization of limit sets. In this paper, we generalize the notion of WAPT to 
set-valued dynamical systems and extend Theorem A to these more general settings. 

The paper is organized as follows. The first step (see Section 2) consists in defining 
properly the notions of invariant measure and Birkhoff center for set-valued dynam- 
ical systems. To this end, we heavily rely on Miller and Akin (1999). In this paper, 
the authors prove the equivalence between various definitions of an invariant mea- 
sure for discrete time set-valued dynamical systems induced by closed relations. We 
give two equivalent definitions in the continuous time case (see Theorem 2.6). In the 
same framework, Aubin, Frankowska and Lasota (1991) prove a Poincare recurrence 
Theorem. We give a topological version of this result in the continuous time case (see 
Theorem 2.10). In Section 3, we prove the main result about the ergodic behavior 
of WAPTs (Theorem 3.2). We then give some examples of WAPTs in Section 4, in 
particular stochastic approximation algorithms relative to a differential inclusion. 
The proofs of several technical results are postponed to the appendix to ease the 
reading. 

2. Ergodic theory for set- valued dynamical systems 

Set-valued dynamical systems, which are often referred to as general control sys- 
tems or general dynamical systems, are generally used to describe multi- valued dif- 
ferential equations, including differential inclusions (see Bianchini and Zecca (1981), 
Li and Zhang (2002), Benaim, Hofbauer and Sorin (2005, 2006) or Nieuwenhuis (2009)) 
and control systems (see Roxin (1965) or Kloeden (1975, 1978)). The literature on 
the subject is abundant and the terminology sometimes differs among authors. 

2. 1 . Generalities 

Let us first recall some classical notions. The Hausdorff distance between two nonempty 
closed sets A and B in E \s given by 

Dh{A,B) ■.= TaB.^{dH{A,B),dH{B,A)] , 

where du is the Hausdorff semidistance: 

dH{A,B) := supd{a,B). 

Let C(M, -E) denote the space of continuous £^-valued applications, endowed with 
the topology of uniform convergence on compacts. This topological space is metriz- 
able with the distance D, given by 

^(x,y) := X] ^^'^{1' '^^P (^i^it),y{t))}, 

km ^ te{-k,k\ 
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which makes it complete (since E is complete). 

Definition 2.1. A set-valued map <I> : x E ^ with nonempty and closed 
values is called a set- valued dynamical system(SVDS) on E provided that 

a) Vx G E, ^>o(2;) = {x}, 

b) Vx G E, ys,t£ R+, M^s{x)) = ^t+six), 

c) for any x ^ E , t ^ <Pt{x) is a continuous map for the Hausdorff distance, 

d) for any t G M+, the map x ^ <Ptix) is upper semicontinuous, i.e. for any 
xq G E, for any e > there exists (5 > such that (i//(<I>j(x), <I>j(2;o)) < e for 
any x G B{xo, 6). 

Definition 2.2. A function z : [0,T] ^ E is a partial solution relative to the set- 
valued dynamical system $ if it satisfies z{t) G <I>t_s(z(s)) for s,t G [0,T], s < t. 
The set of such solutions is called S^''^K We call s]^''^\A) the set of partial solutions 
on [0,T], starting in A. We denote by the set of half solutions S^^'°°\ 

The partial solutions are continuous functions and, given < t and two points 
x,y (z E such that y G ^t(x), there exists at least one partial solution z on [0, t] such 
that z(0) = X and z(t) = y (see Roxin (1965)). By the above remark, for all x (z E, 
s]^'^\x) 7^ which implies that S'^{x) 7^ 0. A function z : M — )• which satisfies 
z{t) G <I>f_s(z(s)), Vs,t G M, s < t, is called an entire solution of ^. is the set of 
all entire solutions, S^{A) the subset of entire solutions starting from A. We will say 
that the set-valued dynamical system $ is complete if, for any x ^ E, there exists 
an entire solution with initial condition x, i.e., if S^{x) 7^ 0, Vx G E. For example, 
we will see in Section 4 that the SVDS induced by a standard differential inclusion 
is complete. 

From now, let us assume that E is compact and call it M to avoid confusions. 
Therefore, we consider a SVDS defined on M. 

Proposition 2.3. The set of solutions is a nonempty compact subset o/S(M, M). 

The compactness is a well-known consequence of Barbashin's Theorem^ (see for 
instance Barbashin (1948) or Aubin and Cellina (1984)). We prove the existence of 
at least one entire solution in the Appendix A. 

Remark 2.4. Points c) andd) in Definition 2.1 imply that the map {t,x) 1— )■ $t(x) is 
jointly upper semicontinuous. Since M is compact, upper semicontinuity is equivalent 
to saying that the graph of^: 

Gr(«>) := {it,x,y) : y G $t(x)} 

is closed: if {tn,Xn,yn) it,x,y) and yn G $t„(x„) then y G $t(x). 

^this means that $ is a map from R+ x £ to 2^. Set-valued maps are also called relations in 
the sequel. 

^Which states that, for any t > 0, the set Sp'^^A) is compact in e([0,t],R) 
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For our purpose, we need to give a proper definition of an invariant measure 
relative to set-valued dynamical systems. Recall that, if = {9t}t€R (resp. 9 = 
{9t}teM.+ ) is a continuous flow (resp. semiflow) on a metric space X, a measure fi on 
X is called 9-invariant if fi{9^^{A)) = fJ.{A) for all Borel set A ^ X and t € R (resp. 
t € R+). 

Let X and X' be two metric spaces and / be a Borel measurable map from X to 
X'. We denote by y(X) the set of probability measures on X. Then we can define the 
map /* : ^'(X) ^ J>{X') by 

rifiKA') ■.= ^,{r\A)), 

for any /u G [P(X) and any Borel set A' in X'. The support of /u G J'(X), i.e. the 
smallest closed set S CX satisfying f^{S) = 1, is denoted by supp(^). In the sequel, 
given a closed subset S of X, we will sometimes assimilate a probability measure 
with support in S to its restriction to the metric space {S, d). 

2.2. Invariant measures for a closed relation 

Let F : M ^ M be a closed relation with nonempty values, which means that the 
graph of F, 

Gr{F) := {{x,y) eM x M \ ye F{x)} 

is closed. Let be the set of bi-infinite sequences in M. The relation F induces 
a discrete time set-valued dynamical system on M, whose set of solutions is the 
nonempty set 

Sf := |x G : x,+i G F(x,) Vi G z} . 

In order to define invariant measures in this discrete case, we follow Miller and Akin 
(1999). The shift homeomorphism Q : — ^ is defined by (6(x)). = Xj+i, 
for z G Z. Notice that M^, equipped with the product topology is metrizable via the 
following distance: 

(5(x,y) := supmin{d(xfc,yfc),l/|A;|} 

kez 

The following theorem is due to Akin and Miller (see Miller and Akin (1999), 
Theorem 3.2). 

Theorem 2.5. A probability measure fi G J'(M) is called an invariant measure for 
F if it satisfies the following equivalent conditions. 

1. For every Borel set A d M 

,,{A)<iJt{F-\A)). 

2. There exists a Markov kernel k : M — > CP(M) satisfying 

X G supp(/i) =^ supp(k(x, •)) C F{x) 

and 

= K*{fi){.) := K{x,-)fi{dx). 
Jm 
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3. There exists jl G J'(M x M) the space of probability measures on M x M such 
that 

supp(/x) C Graph{F) 

and 

where tTj : M x M — t- M is the ith projection. 
4- There exists a probability measure v on which is invariant with respect to 
the shift homeomorphism Q, satisfying 

supp(z/) C Sp 

and 

where ttq : x G xq G M. 

The set Vp^M) of F -invariant measures is nonempty, compact and convex in 
J'(M). In general, if n e ^fIM) then 

supp(/x) C TtoiSp). 

2.3. Invariant measures for set-valued dynamical systems 

For a set- valued dynamical system we now give two equivalent definitions of an 
invariant measure. Notice that, by definition, for any t G M+, the set- valued map 
$j : M ^ M is a closed relation with nonempty images (hence its set of solutions 
5$^ is nonempty). Let us introduce the Lipschitz map ttq : S(M, M) — )• M defined by 
7ro(y) = y(0) and the translation flow 6 : R x e(M, M) -)• e(R, M), which associates 
to a real number t and y G C(R, M) the translated map 0t(y), defined by 

et{y){s)=y{t + s). 

Theorem 2.6. Given a probability measure /x G J'(M), the two following statements 
are equivalent: 

(i) for any t > 0, there exists a probability measure vt on 5$^ such that 

2. vt is Q -invariant, 
(ii) there exists a probability measure v on 5$ such that 

a) ■kI{v) = \x, 

b) V is @ -invariant. 

Such a probability measure will be called an invariant measure for the set-valued 
dynamical system We call y$(M) (or 7^) the set of invariant measures for $ . 
It is a nonempty compact convex subset of J'(M) . 
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Proof. In order to prove (i) (ii), we define, for all t > 0, a new relation 

Kt : Gri^t) ^ 4°'*' 
which associates, to {x,y) E Gr(<I>j), the set 

Kt{x,y) = {z€ 4°'*] : z(0) = x ;z(t) = y}. 

Notice that Kt{x,y) is nonempty and Kt is a closed relation for all t > 0. Indeed, 
assume that the sequence {xn,yn)n converges to {x,y) € Gr{^t) and that, for all n, 
G Kt{xn,yn) a-iid z" — 7> z G S^'^\ We easily have z(0) = x and z{t) = y and so 
z G Kt{x,y). 

Since 4'*^ compact, closedness of Kt is equivalent to upper semicontinuity, 
which can also be stated: for any A closed in s]^'^\ K^^{A) is closed in M x M. 

In particular, Kt is measurable: for any closed A C S^'^\ KfT^^A) is a Borel set. 
By Theorem 8.1.3 of Aubin and Frankowska (2009) we can therefore choose, for all 
t > 0, a measurable selection of Kt, 

Kt : Gr(ci>t) ^4°'*'. 

Let t > be fixed for now. We claim that there exists a measurable application 
ht : 5$ which conjugates the shift operators and 0j: 

ht oQ = Qto ht. (2) 

In order to prove (2), we now define two sets : 

A ■■= {{xn,yn)n€Z G Gr($t)^, | y„ = Xn+1 for ah n}; 

and 

Bt := {(z")„ez € (S^^'Y I ^"+^0) = z"(t)}. 

Notice that S^^ , endowed with the metric 6 (see previous section) is topologically 
equivalent to At, seen as a subset of the product space Gr(<I>f)^, equipped with the 
induced product topology. Similarly, (5$, D) is topologically equivalent to the set Bt, 
understood as a subset of the product space (4'*')^' equipped with the product of 
the uniform convergence topology on [0,t]. We now construct a measurable function 
ht from At to Bt (to which can be associated a measurable function ht from 5"$^ 
to S^) the following way: let (x, y) = {xn,yn)n be in At. Then /ii(x, y) is given by 
z = (z")„,, where 

tP'^s) = Kt{Xn,yn){s), M S £ [0,t]. 

In other terms, ht is the countable product of the measurable map Kt and therefore 
is measurable. 

Now the corresponding map ht : S^^ — > is also measurable. To understand why 
the conjugacy (2) holds, let us give some insights on the map ht: given {xn)n £ S^^, 
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we consider, for each couple {xn,Xn-\-i) its image by Kt (which is a partial solution 
curve of length t) and then build a solution z G 5$ by joining together these partial 
solution curves. The conjugacy is a clear consequence of this construction. 

By assumption, for any t > 0, there exists a probability measure on S^^ such 
that 

2. i/^ is invariant for 0. 

Let {tn)n be a strictly decreasing sequence, converging to and be a limit point of 
{hf^{i't„))n (such a point exists since the considered sequence is tight). Notice that 
u sits on S^. We can assume, without loss of generality, that v = lim^^oo ^tn^^tn)- 
Since irooht = ttq, we have 7rg(i/) = fj,. There remains to prove that is 0-invariant. 
Let / be a continuous function on and T > 0. It is sufficient to prove 

/ f{z)u{dz) = I f{eT{z)Mdz). (3) 



We call 



in = ^ f{z){hlM){dz) and C = y" foQT{z){hlM){dz). 

The two members of equation (3) can be rewritten in the form 

^ := lim and ^' := lim 

Recall that Qt o ht = hi o @ for all t > 0. Call Sn = T — [-r-lin- Then 
Since is 0-invariant, we get : 

i'n = j foes,s^){hi{utj){dz). 

Now we prove that — ^,^[ converges to zero. Pick e > 0. Since 5$ is compact, 
(t,z) i-T- 0t(z) is continuous and s„ 0, there exists large enough so that 

|/oe,„(z)-/(z)|<£, 

for all n > and z S 5$. Since is a probability measure for all n, we get the 
equation (3). 

Conversely, assume that (ii) holds. There exists a probability measure v which 
satisfies a) and b). For any t > we define the application gt : — > S^^ which 
associates to z, gtiz) = {z{kt))k^z and another probability measure 
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Since ttq ° 5* = '''"Oi we have TTQ{i't) = ^^{gKi')). Now we show that i^t is ©-invariant. 
Since @ o = o Q^^ we have 

e*(^,) = e*(5;H) = (eo5,)* 

The set lPe|s of ©[5^ -invariant probabiUty measures is a convex and compact 
nonempty set (by Krylov-Bogolubov Theorem). Since y$ is the image of ^Q^g under 
tTq, we easily check that it enjoys the same properties. ■ 

Remark 2.7. If ^ = (f) is a flow on M then tto restricts to an homeomorphism of Scf, 
to M. Consequently, /i is cp-invariant if and only if it is the image of a Q-invariant 
measure ( with its support in ) under ttq . 

Remark 2.8. Another natural way to define invariant measures relative to a SVDS 
is to consider the set of half solutions 5^ instead of 5$. Let us define the map 
0+ : M+ X e(M+,M) e(M+,M) as the shift semi-flow defined similarly to 0. 
A probability measure ^ on M is then said to be a semi-invariant measure for the 
set-valued dynamical system ^ if there exists a probability measure on C(M-|-,M) 
such that 

(i) supp(i/+) C S"^, 
(a) is -invariant, and 

(ill) {tt+)*{u+) = fi, where tt+ : e(R+, M) ^ M, 7r+(y) = y(0). 

Semi-invariance is, a priori, very similar to invariance (as defined in Theorem 
2.6). We discuss the relationship between these two definitions in Appendix C. 

2.4- Poincare recurrence theorem for set-valued dynamical systems 

As an apphcation of the definition of an invariant measure, we shah state a topo- 
logical version of the Poincare recurrence theorem for set- valued dynamical systems. 
Roughly speaking, this theorem says that invariant measures sit on the closure of 
the set of recurrent points of the dynamic. The concept of recurrent point is closely 
related to the notion of uj-limit set of a point x € M, defined by 

:= ^[t,oo){x) 

It is characterized by the following: y € uj^{x) if and only if there exists (tn)n t 
00, (z")„ C S^{x) such that z"'{tn) y (see Benaim, Hofbauer and Sorin (2005)). 
Notice that a;$(x) contains the limit sets L(z) = Q^^q z([t, 00)) of all solutions z 
with z(0) = X but is in general larger than 

L{x):= U Liz). 
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(See Benai'm, Hofbauer and Sorin (2005)) 

In the classical framework of a flow, a point is recurrent provided it belongs to its 
own w-limit set and the topological version of Poincare recurrence theorem is stated 
as follows 

Theorem 2.9 (Poincare). Let (X,d) be a separable metric space and 9 = {Ot}t£M. o- 
flow on X. Let ji be an invariant measure for 9. The closure of the set of all recurrent 
points, 

:= {x G X: X E a;e(x)}, 
is called the Birkhoff center of 6 and denoted BC(0). Then 

^^{BC{e)) = 1. 

The same definition of recurrence does not seem to fit in the set- valued framework. 
Intuitively, a point x is recurrent if there exists an entire solution starting from x, 
whose limit set contains x. Thus it is more natural to define the set of recurrent 
points of ^> by 

i?$ := {x G M I X G L{x)}. 

Clearly we have ^<j, C := {x G M [ x G uj^{x)}. The closure of will be 
called the Birkhoff center of <I> and will be noted BC(<I>). Notice that, if $ is actually 
a flow, then L{x) = u{x) and consequently i?^ = The following statement is a 
Poincare recurrence theorem for set-valued dynamical system. 

Theorem 2.10. Let (i be an invariant measure for then 

^x{Bcm = 1. 

Proof. Let fi be an invariant measure for <I> and v be an invariant measure for 0|g 
such that ttq{i') = First of all, notice that 

7ro(BC(e)) C BC(«>). 

Indeed, pick z G i?n . There exists a sequence t„ f +oo such that Qt (z) -^n z. In 

particular, 7ro(z) = z(0) = lim„z(t„), which means that 7ro(z) G L(z(0)). Using the 
last inclusion, we get 

= Kvro"' o vro(BC(e))) 
> i/(BC(G)). 

The last quantity is equal to one by Theorem 2.9. ■ 
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3. Ergodic properties of weak asymptotic pseudotrajectories 
3. 1 . Definition 

Let (O, P) be a probability space and {3^t ■ t > 0} a nondecreasing family of 
sub-(7-algebras. We define a process 

X : R+ X — > M, 

to be a weak asymptotic pseudotrajectory (WAPT) for the SVDS $ if it enjoys the 
following properties 

(i) for almost every uj, the path X(-,aj) is miiformly continuous; 

(ii) it is progressively measurable : X|jo,r]xn is '^[o,T] ^ measurable for all T > 0; 

(iii) for each a > and T > 0, we have 



lim P inf sup d (X(i + s), z(s)) > a 



S^t =0 



almost surely. 
3.2. Ergodic behavior 

Given a weak asymptotic pseudotrajectory X for $ and ujE.fl, let lJ,t{oj) denote the 
empirical occupation measure of X(-,a;): 

t 



1 r 

r Jo 



Remark 3.1. //t(w) is defined as the unique Borel measure on M such that, for all 
continuous function f on M , 



\ f f{Xis,uj))ds= [ f{x)iit{u){dx) 



Let J'(x,c<;) <^ ^CM) denote the set of weak* limit points of {lJ.t{oj)}t>o- Notice 
that, since M is compact, J'(x,a)) nonempty and compact. We now state the main 
result of this section: 

Theorem 3.2. Let X. be a weak asymptotic pseudotrajectory for the set-valued dy- 
namical system Then 

•P(x,w) ^ -P*) almost surely. 

In particular, 

supp(/x) C BC($), almost surely. 

Me?(x,u,) 
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Before proving this theorem, we state some useful lemmas. First we introduce a 
family of probability measures {t't}t>o on C(R, M) related to the family {/it}t>o : 
given (J G ri, 

1 /■* 

M^) ■= 7 / '5e,(x(-,.^))C^'S, 
^ Jo 

where we use the convention that an element X S C(IR+,M) can also be seen as an 
element of e(R, M) with X{t) = X(0) for all t < 0. 

Lemma 3.3. The set {0f(X(-,a;)) : t > 0} is almost surely relatively compact. 

Proof. Hypothesis (i) in the definition of a WAPT guarantees that the set of func- 
tions {0((X(-,a;)) : t > 0} is almost surely equicontinuous. Since X(M, w) C M 
(which is compact), we can apply Ascoli-Arzela's theorem to conclude. ■ 

Lemma 3.4. The family {i^t{^)}t>o is almost surely tight. 

Proof. By Lemma 3.3, we know that {0j(X(-,a;)) : t > 0} is almost surely relatively 
compact. The tightness follows since the support of is included in {0f(X(-, w)) : t > 0}, 
for any t >0. ■ 

The last lemma is a generalization (in the continuous case) of Theorem 6.9 in 
Walters (2000). A short proof is provided in appendix B for convenience. 

Lemma 3.5. Let {X,d) be a compact metric space, 9 = {9t)t£R be a flow on X and 
{(^t}t>o be a collection of probability measures on X. Consider the family {i't}t>o of 
probability measures on X, defined by 

yt = - fei{at)ds. 
1^ Jo 

Then any limit point v of {yt)t>o is 9-invariant. 
We are now ready to prove our main result. 

Proof of Theorem 3.2. 

For all Lo G Q and for all /i G ^'(x,^)) there exist {tj)j>o going to infinity and a 
probability measure v on C(IR, M) such that 

1. fj,tj fi and i^tj V, 

2. V \s 0-invariant, 

3. 7r*(i/) = 

The first point is a direct consequence of the definition of /i and the tightness of 
{^t}t (see Lemma 3.4), the second point is a consequence of Lemma 3.5 and the last 
point follows from the continuity of the map ttq and the fact that 7rg(ft) = /i^, for 
all t G M+. The set of all such u will be called A{(jj, /x): 

A{uj,^) = {v ^ y(e(M,M)) : 3tj t oo such that l.,2. and 3. hold} 

Let Aibj) = U^gy(x^j^(a;,/x). We have A{ijj) C ^6(6(1^,^)), the set of ©-invariant 
probability measures on C(M, M). We now exhibit a set O C of full measure such 
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that for all € r2 and for all i' E A{u;), supp(j^) C 5"$. Let {Cfc} be the family of 
closed neighborhoods of defined by 

Cfe = {z € e(M,M) : D(z,S<s>) < 

k 

It is sufficient to find, for all A; > 0, a set 0^ C of full measure such that for all 

IJ supp(z^) C Ck- 

ueA{u]) 

Let /c S N and G N (large) be fixed. First of all we choose T > such that 

T/N G N and YIJLt/n 'h ^ Ik' (5 > be small enough such that 5 Yl^Lo ^ < 'ik- 
With these choices of T and 6, we have for all y G C(R, M) and s > 0, 

sup d(e,(X)(n),y(u)) <(5 D{esCS.),y) <y. 

u&[-T/N,T/N] 1^ 
Therefore we have for alH > 1, 

sup d(e(,_i)r(X)(«),y(^z)) <(5 ^ I?(e.(X),e,_(,_i)^(y)) <i (4) 

for all s G [(i — 1 + 1/N)T, {i — \/N)T]. Since is invariant for 0, (4) implies that 
the event 



inf sup (i(G(i_i)r(X)(u),z(n)) < (5 



is contained in the event 



mj^ D (e,(x), z) < i, G [(i - 1 + ^)r, (i - l)r] [> . (5) 



For n > 1 set 
and 



— I{inf^gs^ sup„6[0,Tl '^(0{«-i)t(X){«),z(u))><5} 
n ^ 

Since M„ is a martingale and sup„E(M^) < 4^ t^, Doob's convergence theorem 
implies that (M„) converges almost surely. Hence, by Kronecker lemma, 

lim - y^{Ui - E(C/,| = (6) 

i=l 
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almost surely. In others terms, there exists flk,N C (a subset of full measure) such 
that for all cj G ^k,Ni (6) holds. Now pick to € i^k,N E^nd E A{uj). Let rij = [tj/T]. 
Then 

1 ^ r{i-l/N)T 

u{Ck) > lim — > / 6e,(x){Ck)ds 

> (1 - 2/iV) lim - rj.,, / <5e4x)(Cfc)ds 

1 \ 

> (1 - 2/N) lim — ^ I{inf,gs^ D(e4X),z)<l, Vs e [(i-l+l/7V)T,(i-l/iV)T]} 

^ i=l 



> (1 -2/iV) ( 1 -lim — ^C/i 



i=l 



1 - hm - ^ (t/, - E(C/,| + 4^ E(C/,| 

The first term in the last equality is equal to zero. Additionally, by definition of a 
WAPT, we may assume without loss of generality that, on ^Ik N, 



lim F inf sup d{X{t + u) , z{u)) >5 S't] =0- (7) 

Consequently, v{Ck) > 1 — 2/N for all v G \J^<z^^, j^A(oj)^ which means that 
v{Ck) = 1 for all V € \Ji^^^^A{uj), with 0^ := fliv^^fc.iv- We conclude the proof by 
setting (l = VLk- ■ 

3.3. A simple deterministic example 

Notice that the main result of previous section is not useless in the case of APTs. 
To understand why, let us consider a set- valued dynamical system on the circle S^, 
defined by the differential inclusion ^ G F{x) with 

^ ' \l-x if < X < 1. 
The dynamics have the following portrait 
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We easily check that the only two internally chain transitive sets are {0} and S^. 
Let X be a bounded APT of this dynamics. Then by Theorem 4.3 of Benai'm, Hofbauer and Sorin 
(2005) we know that L{X) is either {0} or S^. Moreover the only invariant measure 
for the dynamic is the Dirac measure on 0. Therefore by Theorem 3.2 we know that 
any limit point of the empirical measure of X is 5q. In particular, this means that 
X spends most of its time near 0. 

4. Weak perturbed solution of a difTerential Inclusion 

In this section, we give some examples of WAPTs relative to a particular case of 
SVDS: the set-valued dynamical systems induced by a differential inclusion. In the 
whole section, we are in the particular case where E is the Euclidian space M"^. 

Definition 4.1. A set-valued map F : M™" ^ M"* is said to be standard if it satisfies 
the following assumptions: 

(z) for any x € M™', F'ix) is a nonempty, compact and convex subset o/M™, 
(ii) F is upper semicontinuous (see Definition 2.1), 
(Hi) there exists c > such that 

sup ll^ll < c(l + |tx||). 

zeF{x) 

Under the above assumptions (Definition 4.1), it is well known (see Aubin and Cellina 
(1984)) that the differential inclusion 

I^^W (8) 

admits at least one solution (i.e. an absolutely continuous mapping z : M ^ such 
that z{t) G F{z{t)) for almost every t) through any initial point. To any x G and 
t € M-)-, we can therefore associate the nonempty set 

^t{x) '■= {z(i) [ z is a solution of (8), z(0) = x} . 

It is not hard to check that = (^'t)tgR is a complete set-valued dynamical system 
(see e.g. Benai'm, Hofbauer and Sorin (2005)). 
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Definition 4.2. A set M C is invariant for ^ if, for every x ^ A, there exists 
an entire solution curve z such that z(R) C M . 

Let M C M™" be a compact and invariant subset (for <l>). In tlie following we will 
consider the complete SVDS restricted to M, that we will also write $ and that is 
defined, for all x G M and t € ]R-|_, by 

^t{x) := {z(t) I z is a solution of (8), z(0) = x, z(M) C M} . 

Given a positive number 5, let be the set-valued map defined by 

F\x) := {y\ 3z£ B{x,6) such that d{y,F{z)) < 6} . (9) 

Definition 4.3. Given a function S :]0, +oo) — )• [0,1] decreasing to zero as t goes 
to infinity and a locally integrahle process U : M+ x ^ M™", we say that a process 
Y : X — )• M is a (d, \J)-weak perturbed solution of the differential inclusion (8) 
provided 

{i) Y is absolutely continuous for all u, 
{a) for almost every t > 0, 

^-U(t)€F^W(Y(t)), 



{Hi) for any T > and any 7 > 0, 



lim ] 





rt+s 




sup 


J U(n)dn 




V^e[o,T] 







0, 



almost surely. 



Theorem 4.4. Assume that 'Y is a {S,'U)-weak perturbed solution of the differ- 
ential inclusion (8) and that U is uniformly hounded by a positive constant C: 
sup^gQ supjgjQ^^] U(t, w) < C. Then Y is a weak asymptotic pseudotrajectory of 

Proof. Let T > and define \\F\\ := sup^eM ^^PyeF{x) WuW < Consider the 
compact set 

K := {y G Lip{[0,T],R"') \ Lip{y) < ||F|| + C + 1 , y(0) G M} , 

where Lip{[0,T],M"^) denotes the set of Lipschitz functions on [0, T] and Lip{y) is 
the Lipschitz constant of y. The set K is well adapted to our problem because it 
contains every solution curve of (8), restricted to an interval of length T and every 
realization of any (S, U)-weak perturbed solution of the differential inclusion. 
For 6 € [0, 1], let us define the set- valued application (with the convention A'^ = A): 

-.K^K, z ^ A^(z), (10) 
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where y E A'^(z) if and only if there exists an integrable function h : [0,T] 
such that h{u) G F^{z{u)) Vn G [0,r] and 



y(r) = z(0) + / h{u)du, Vr G [0,T]. 
Jo 



Notice that Fix{A) := {z G K \ z G A(z)} is equal to S^' , the set of partial solu- 
tions of $ on [0, T]. 
By (m), we have 

^^^1^ - V{t) G F^^^'\Y{t)), for almost every t > 0. 



There exists an integrable function h : [0, T] 
u)) Vn G [0,r] and, for any r G [0,T], 



such that G F^W(Y(t + 



Y(t + r) 



i+r 



V{u)du = Y{t) + / h{u)du 



Hence3, d[o,T] (Y(t + ■),A"^'\Y{t + •)) < sup,g[o,T] It U(^)dn 

Let a > 0. The following statement is an immediate consequence of Corollary 
4.11 in Faure and Roth (2010): there exists 7 > (which depends on T and a) and 
^0 > such that, for any 6 < Sq 

rf[o,T](z, A'^(z)) < 7 ^ dio^T]{z,S^) < a. 
Consequently, for t large enough, 

d[o,T] (Y(t + -),S^)>a^ d[o,r] (Y(t + •), A'^*^ (Y(t + •))) > 7- 
For these choices of t and 7, 

P(d[o,T](Y(t + -),5<i.) >a < P(d[0,r] (Y(t + -),A''W(Y(t + -))) > 7 I ^^t) 









< I 


^ sup 


y ij{u)du 




V^e[o,T] 





By (iii), the last term tends to zero when t goes to infinity and the proof is complete. 



4.1. Stochastic approximation algorithms 

Stochastic approximation algorithms were born in the early 50s through the work 
of Robbins and Monro (1951) and Kiefer and Wolfowitz (1952). Let F : ^ 
be a standard set-valued map and M C M"^ be a compact subset invariant for the 
set-valued dynamical system induced by the differential inclusion (8). 

^we call d[o,T] the uniform distance on [0, T] 
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Definition 4.5 (Weak generalized stochastic approximation process). Let {Un)n be 
an uniformly bounded MJ^ -valued random process and {Fn)n sequence of set-valued 
maps on M™. We say that {xn)n is a generalized stochastic approximation process 
relative to the standard set-valued map F on M if the following assumptions are 
satisfied: 

(i) we have the recursive formula 

) J 

{a) the step size {'^n)n is deterministic and satisfies 

E7„ = +00, lim7„ = 0, 
n 

n 

{Hi) for all > 0, G M , 
{iv) for all T > and all 7 > 0, 



lim P sup ' 



k-1 



k-1 



k such that 7i < 7" / > 7 



T 



0.(11) 



(f ) for any 6 > there exists no € N such that 

Mn > HQ, Fn{Xn) C F ( 

Remark 4.6. Let {Un)n be a M"^-valued random process adapted to the filtration 
(3"n)n such that 

(i) E([/„+i|5-„) = 0; 
(a) for all T > 0, we have 

liin supE (||?7„+i|| l|||c7„_^^||>if}|3'„) =0. 

Then for allT > and all'j > 0, Property (11) is satisfied (see Benaim and Schreiher 
(2000)). 

Consider a weak generalized stochastic approximation process (x„)„. Set r„ := 
Sr=i 7* ^^^^ m{t) := sup{j I Tj < t}. We call X the continuous time affine in- 
terpolated process induced by (x„)„, 7 the piecewise constant deterministic process 
induced by (7n)n and U is the piecewise constant continuous time process associated 

to {Un)n- 



X(rj + s) = Xj + s— -, for s G [0, 7j+i] ^{n + s) := ji+i for s G [0, 7j+i[, 

7i+i 



and U{t) := f7„+i, for t G [r„,r„+i]. 
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Theorem 4.7. The interpolated process X is a WAPT. Hence the conclusions of 
Theorem 3.2 hold. 

Proof. By straightforward computations (see the proof of proposition 1.3 in Benaim 
et al. Benaim, Hofbauer and Sorin (2005)), it is not difficult to see that (X(t))t is a 
weak perturbed solution associated to U and 

5{t) := inf {5 > I r„ > t ^ F„(x„) C F'^(x„)}+7(t) {v{t) + c[l+ sup F{x)^^ 

which converges to 0. Consequently X is a WAPT relative to the SVDS induced by 
F and the proof is complete. ■ 

Appendix A: Proof of Proposition 2.3 

Recall that an element z G C(M_|_,M) can also be seen as an element of C(M, M), 
with the convention z(t) = z(0) for all t < 0. Since S'^ 7^ 0, the existence of at least 
one entire solution is a direct consequence of the following Proposition. 

Proposition A.l. Let (tn) 'I 00 be a sequence of positive real numbers converging 
to infinity and z G S'^ be a solution. Then there exists a subsequence (tnfe)fc>o '^f^d 
an entire solution z € such that 

lim ©tn. (z) = z. 
fe— ^00 

Proof. Pick some positive integer A^. By Barbashin Theorem, the set s]^ ^'^^ is 
compact and, for n € N large enough, 0f^(z) € Therefore there exist an 

increasing sequence ip^ : N — > N and E s]^ ^'^^ such that 

lim e. ,(z) = 2^^, 

By the same arguments, there exist an extraction tp and z^"*""*^ G S'jj, ^ ii^+i] g^^j^ 
that 

lim ^ (z) = z^+\ 

and then, in particular, z|^^^j = z^ . We set ip^^'^ ■= ip o ip^ and we iterate the 
process. In this way we construct an entire solution z G 5$ such that Z|[_7v,Ar] = 
z^ ViV. 

Let ((5fc) 4, be a decreasing sequence of positive real numbers. There exists a 
natural number mi such that for all m > mi, 

sup d(ej (z)(s),z(s))<(5i, 

se[-i,i] ^ ^ ' 

We set Hi := ip^[mi). Now we define Uk by induction. Fix k > 1. There exists a 
natural number > mk-i such that for all m > ruk, 

sup d{Qt ii)is),z{s)) < 6k, 

s£[-k,k] 
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We set rifc := ^*'(mfc). Therefore, by construction, we have hm^^oo ©*„ (z) = z. ■ 



Appendix B: Proof of Lemma 3.5 



Let fi he a hmit point of {fJ.t}t>o- there exists a sequence {tn)n t +c« such that 
IJ, = HiUn-i-oo fJ'tn- We have to prove that, for any T and any continuous function / , 



fix)dflix) = / fieTix))dfiix). 



(12) 



We have 



f{x) dfi{x) = lim — 

tn Jo Jx 



f{es{x))dat„ix)ds 



and 



/ f{0Tix))d^i{. 

Jx 



x) = lim — 

" tn Jo JX 



f{es+T{x))dat„{x)ds. 



By Fubini's Theorem we can exchange the integral operators in both expressions. 
Consequently, 



1 



JX 



fi9six))ds)dat,Ax) 



JX 



f{es+Tix))ds)dat„{x) 



< 



< 



< 



X Jo 



f{es{x))ds)datAx) 



X Jo 



f{es+T{x))ds)dat„{x] 



1 

tn JX 

2T\\f\ 



fiOsix))ds 



+ 



tn+T 



f{0s{x))ds 



d(Tt„ ix) 



tr, 



Finally, taking the limit as n goes to infinity, we obtain (12). 



Appendix C: Some remarks on semi-invariance 



In this section we show (Proposition C.l) that every invariant measure for a SVDS 
<^ is a semi-invariant measure for <I>. To do this, we start by proving two technical 
lemmas. Let us define the projection y'*' on C(M+, M) of an element y G C(R, M) by 

y+{t)=y{t), >0, 

and, in the same way, A"^ := {y+ : y G ^4} for a subset A C C(M, M). Be aware 
that the set {S^)~^ is contained in the set S"^ of partial solutions, but not equal in 
general. 

Proposition C.l. An invariant measure fj, on M for the set-valued dynamical sys- 
tem ^ is a semi-invariant measure for <I>. 
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Proof. There exists a probability measure on such that 

a) 7r*{u) = n, 

b) V is 0-invariant. 

We need to construct a probabihty measure on C(IR+,M) which satisfies condi- 
tions (i), (ii) and (iii) of the definition of a semi-invariant measure given in Remark 
2.8. A natural way to do this is to define as 

z^+(yl) := v{Ba), for all Borel sets A of e(M+,M), 

where Ba '■= {z € /S^ : € A}. First, we have 

i^+{St) = ({z G 5$ : z+ G 5+}) 
= z^(5$) = l. 

which gives condition (i). Let T > and A C a Borel set. Since, for any z G 5$, 
we have e+(z+) = (er(z))+, 

B{e+)-HA) = {zG5$:z+G(e+)-i(A)} 
= {z G 5$ : (er(z))+ G ^} 

Therefore, as v sits on 5$ and is ©-invariant, we have 

u+{{e+)-\A))=u^B^Q+^.,^^^)=u{@^\BA))=HBA) = 'y^{A), 

which gives condition (ii). 

Let D he a Borel subset of M. Notice that = ttq^{D) (1 = S^{D). 

Consequently 

= u+{i7r+)-\D)) = ,^iS^m = K^o H^)) = f^iD) 

and the result holds. ■ 

Whether the converse statement is also true is an open question. 
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